#############################
University of Cambridge Computer Laboratory
-30 Jan 2004
+15 Jun 2004
http://www.cl.cam.ac.uk/netos/xen
investigate interesting research issues regarding the best techniques
for virtualizing resources such as the CPU, memory, disk and network.
The project has been bolstered by support from Intel Research
-Cambridge, who are now working closely with us. We're also in receipt
-of support from Microsoft Research Cambridge to port Windows XP to
-run on Xen.
+Cambridge, and HP Labs, who are now working closely with us. We're
+also in receipt of support from Microsoft Research Cambridge to port
+Windows XP to run on Xen.
Xen enables multiple operating system images to execute concurrently
on the same hardware with very low performance overhead --- much lower
and performance is contained in our October 2003 SOSP paper, available
at http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf
-We have worked on porting 3 different operating systems to run
-on Xen: Linux 2.4, Windows XP, and NetBSD.
+We have worked on porting 4 different operating systems to run
+on Xen: Linux 2.4/2.6, Windows XP, NetBSD and FreeBSD.
-The Linux 2.4 port (currently Linux 2.4.22) works very well -- we
+The Linux 2.4 port (currently Linux 2.4.26) works very well -- we
regularly use it to host complex applications such as PostgreSQL,
Apache, BK servers etc. It runs every user-space applications we've
tried. We refer to our version of Linux ported to run on Xen as
-"XenoLinux", although really it's just standard Linux ported to a new
+"XenLinux", although really it's just standard Linux ported to a new
virtual CPU architecture that we call xen-x86.
NetBSD has been ported to Xen by Christian Limpach, and will hopefully
issues with product activation in this environment which need to be
thought through.
-So, for the moment, you only get to run multiple copies of Linux on
+So, for the moment, you only get to run Linux 2.4/2.6 and NetBSD on
Xen, but we hope this will change before too long. Even running
multiple copies of the same OS can be very useful, as it provides a
means of containing faults to one OS image, and also for providing
Pentium II, Pentium III, Pentium IV, Xeon, AMD Athlon, AMD Duron).
Multiprocessor machines are supported, and we also have basic support
for HyperThreading (SMT), although this remains a topic for ongoing
-research. We're also looking at an AMD x86_64 port (though it should
+research. We're also working on an AMD x86_64 port (though Xen should
run on Opterons in 32-bit mode just fine).
Xen can currently use up to 4GB of memory. It's possible for x86
difficult, but we'd also need to add support to all the guest
OSs. Volunteers welcome!
-We currently support a relatively modern set of network cards: Intel
-e1000, Broadcom BCM 57xx (tg3), 3COM 3c905 (3c59x). Adding support for
-other NICs that support hardware DMA scatter/gather from half-word
-aligned addresses is relatively straightforward, by porting the
-equivalent Linux driver. Drivers for a number of other older cards
-have recently been added [pcnet32, e100, tulip], but these are not
-recommended since they require extra packet copies.
+In contrast to previous Xen versions, in Xen 2.0 device drivers run
+within a privileged guest OS rather than within Xen itself. This means
+that we should be compatible with the full set of device hardware
+supported by Linux. The default XenLinux build contains support for
+relatively modern server-class network and disk hardware, but you can
+add suppport for other hardware by configuring your XenLinux kernel in
+the normal way (e.g. "make xconfig").
-Building Xen and XenoLinux
-==========================
+Building Xen and XenLinux
+=========================
-The public master BK repository for the 1.2 release lives at:
-bk://xen.bkbits.net/xeno-1.2.bk
+The public master BK repository for the 2.0 release lives at:
+bk://xen.bkbits.net/xeno-2.0.bk
To fetch a local copy, install the BitKeeper tools, then run:
-'bk clone bk://xen.bkbits.net/xeno-1.2.bk'
-
-To see how to build Xen, Xenolinux, and all the control tools, inspect
-the tools/misc/xen-clone script in the BK repository (also available
-via http://www.cl.cam.ac.uk/netos/xen/readmes/xen-clone). This script
-can be used to clone the repository and perform a full build.
-
-The build procedure for xenolinux is slightly complicated as its done
-by running the 'mkbuildtree' script over a pristine Linux tree to turn
-it into a xenolinux tree by adding the 'xen' architecture.
-
-There's also a recent pre-built source tree on the project downloads
-page: http://www.cl.cam.ac.uk/netos/xen/downloads/xeno-1.2.tar.gz
-
+'bk clone bk://xen.bkbits.net/xeno-2.0.bk'
+
+You can do a complete build of Xen, the control tools, and the
+XenLinux kernel images with "make world". This can take 10 minutes
+even on a fast machine. If you're on an SMP machine you may wish to
+give the '-j4' argument to make to get a parallel build. You should
+end up with all the binaries and images being placed in the ./install
+directory tree. You can then install everything to the standard
+system directories (e.g. /boot, /usr/bin, /usr/lib/python/ etc) by
+taping "make install".
+
+
+Inspect the Makefule if you want to see what goes on during a
+build. Building Xen and the tools is straightforward, but XenLinux is
+more complicated. The makefile needs a 'pristine' linux kernel tree
+which it will then add the Xen architecture files to. You can tell the
+makefile the location of the appropriate linux compressed tar file by
+setting the LINUX_SRC environment variable
+(e.g. "LINUX_SRC=/tmp/linux-2.4.26.tar.gz make world") or by placing
+the tar file somewhere in the search path of LINUX_SRC_PATH which
+defaults to ".:..". If the makefile can't find a suitable kernel tar
+file it attempts to download it from kernel.org, but this won't work
+if you're behind a firewall.
+
+After untaring the pristine kernel tree, the makefile uses the
+'mkbuildtree' script to add the Xen patches the kernel. "make world"
+then build two different XenLinux images, one with a "-xen0" extension
+which contains hardware device drivers and is intended to be used in
+the first virtual machine ("domain 0"), and one with a "-xenU"
+extension that just contains virtual-device drivers. The latter can be
+used for all non hardware privileged domains, and is substantially
+smaller than the other kernel with its selection of hardware drivers.
+
+If you don't want to use bitkeeper to download the source, you can
+download prebuilt binaries and src tar balls from the project
+downloads page: http://www.cl.cam.ac.uk/netos/xen/downloads/
Using the domain control tools
==============================
=================
This CD is a standalone demo of the Xen Virtual Machine Monitor (VMM)
-and Linux-2.4 OS port (XenoLinux). It runs entirely off the CD,
+and Linux-2.4 OS port (Xenlinux). It runs entirely off the CD,
without requiring hard disk installation. This is achieved using a RAM
disk to store mutable file system data while using the CD for
-everything else. The CD can also be used for installing Xen/XenoLinux
+everything else. The CD can also be used for installing Xen/Xenlinux
to disk, and includes a source code snapshot along with all of the
tools required to build it.
Booting the CD
==============
-The Xen VMM is currently fairly h/w specific, but porting new device
-drivers is relatively straightforward thanks to Xen's Linux driver
-compatibility layer. The current snapshot supports the following
-hardware:
+It should be possible to get Xen working with any relatively modern
+hardware supported by standard Linux. However, the version of XenLinux
+built for the DemoCD is fairly h/w specific. If you need other
+hardware, you'll have to configure and build your own xenlinux kernel.
+Xen does require an 'i686'-class CPU or newer, so won't work on 486's
+or plain Pentiums.
+
+We have compiled in drivers for the following hardware:
CPU: Pentium Pro/II/III/IV/Xeon, Athlon (i.e. P6 or newer) SMP supported
IDE: Intel PIIX chipset, others will be PIO only (slow)
SCSI: Adaptec / Dell PERC Raid (aacraid), fusion MPT, megaraid, Adaptec aic7xxx
Net: Recommended: Intel e1000, Broadcom BCM57xx (tg3), 3c905 (3c59x)
- Working, but require extra copies : pcnet32, Intel e100, tulip
+ Also supported: pcnet32, Intel e100, tulip
Because of the demo CD's use of RAM disks, make sure you have plenty
of RAM (256MB+).
configuration to do this), then select one of the four boot options
from the Grub menu:
- Xen / linux-2.4.24
- Xen / linux-2.4.24 using cmdline IP configuration
- Xen / linux-2.4.24 in "safe mode"
+ Xen / linux-2.4.26
+ Xen / linux-2.4.26 using cmdline IP configuration
+ Xen / linux-2.4.26 in "safe mode"
linux-2.4.22
The last option is a plain linux kernel that runs on the bare machine,
the grub menu, then edit the "ip=" parameters to reflect your setup
e.g. "ip=<ipaddr>::<gateway>:<netmask>::eth0:off". It shouldn't be
necessary to set either the nfs server or hostname
-parameters. Alternatively, once XenoLinux has booted you can login and
+parameters. Alternatively, once Xenlinux has booted you can login and
setup networking with 'dhclient' or 'ifconfig' and 'route' in the
normal way.
sequentially for subsequent domains unless told otherwise.
After selecting the kernel to boot, stand back and watch Xen boot,
-closely followed by "domain 0" running the XenoLinux kernel. The boot
+closely followed by "domain 0" running the Xenlinux kernel. The boot
messages can also sent to the serial line by specifying the baud rate
on the Xen cmdline (e.g., 'com1=9600,8n1'); this can be very useful
for debugging should anything important scroll off the screen. Xen's
because of the poor random access performance of CD drives, the
machine will feel very slugish, and you may run out of memory if you
make significant modifications to the ramfs filesystem -- for the full
-experience, install a Xen and XenoLinux image on you hard drive :-)
+experience, install a Xen and Xenlinux image on you hard drive :-)
You can configure networking, either with 'dhclient' or manually via
'ifconfig' and 'route', remembering to edit /etc/resolv.conf if you
Apache, PostgreSQL, Mozilla etc in the normal way, but because
everything is running off CD the performance will be very sluggish and
you may run out of memory for the 'tmpfs' file system. You may wish
-to go ahead and install Xen/XenoLinux on your hard drive, either
-dropping Xen and the XenoLinux kernel down onto a pre-existing Linux
+to go ahead and install Xen/Xenlinux on your hard drive, either
+dropping Xen and the Xenlinux kernel down onto a pre-existing Linux
distribution, or using the file systems from the CD (which are based
on RH9). See the installation instructions later in this document.
work with Xen, and you'll only be able to configure a VGA X
server. We're working on a fix for this for the next release.
-If you want to browse the Xen / XenoLinux source, it's all located
-under /usr/local/src/xeno-1.2, complete with BitKeeper
+If you want to browse the Xen / Xenlinux source, it's all located
+under /usr/local/src/xeno-1.3, complete with BitKeeper
repository. We've also included source code and configuration
information for the various benchmarks we used in the SOSP paper.
Starting other domains
======================
-Xen's privileged control interfaces can be accessed using a C library
-(libxc.so) or an easier-to-use Python wrapper module (Xc). Example
-script templates are provided in tools/examples/.
-
-Abyway, the first thing to do is to set up a window in which you will
-receive console output from other domains. Console output will arrive
-as UDP packets destined for 169.254.1.0. The DemoCD's startup scripts
-automatically bring up 169.254.1.0 as an alias called eth0:xen (see
-/etc/sysconfig/network-scripts/ifcfg-eth0 )
+The first thing you need to do is to start the "xend" control daemon
+with "xend start". You may wish to add an appropriate link to xend in
+you /etc/rcX.d directory e.g. "ln -sf ../init.d/xend S97xend"
If you're not intending to configure the new domain with an IP address
on your LAN, then you'll probably want to use NAT. The
'xen_nat_enable' installs a few useful iptables rules into domain0 to
-enable NAT. [ NB: The intention is that in future Xen will do NAT
-itsel (actually RSIP), but this is part of a larger work package that
-isn't ready to release.]
+enable NAT. [NB: We plan to support RSIP in future]
Xen has a management interface that can be manipulated from domain0 to
create new domains, control their CPU, network and memory resource
[root@xendemo]# xc_dom_create.py -D ip=commando-1.xeno,169.254.2.3 -Dmem=100
Parsing config file 'defaults'
- VM image : "/boot/xenolinux.gz"
+ VM image : "/boot/xenlinux.gz"
VM ramdisk : "/boot/initrd.gz"
VM memory (MB) : "100"
VM IP address(es) : "128.232.38.51:169.254.2.3"
The purpose of the Demo CD is to distribute a snapshot of Xen's
source, and simultaneously provide a convenient means for enabling
people to get experience playing with Xen without needing to install
-it on their hard drive. If you decide to install Xen/XenoLinux you can
+it on their hard drive. If you decide to install Xen/Xenlinux you can
do so simply by following the installation instructions below -- which
essentially involves copying the contents of the CD on to a suitably
formated disk partition, and then installing or updating the Grub
Because of the use of a RAM-based file system for root, you'll need
plenty of memory to run this CD -- something like 96MB per VM. This is
-not a restriction of Xen : once you've installed Xen, XenoLinux and
+not a restriction of Xen : once you've installed Xen, Xenlinux and
the file system images on your hard drive you'll find you can boot VMs
in just a few MBs.
-The CD contains a snapshot of the Xen and XenoLinux code base that we
+The CD contains a snapshot of the Xen and Xenlinux code base that we
believe to be pretty stable, but lacks some of the features that are
currently still work in progress e.g. OS suspend/resume to disk, and
various memory management enhancements to provide fast inter-OS
Installing from the CD
======================
-If you're installing Xen/XenoLinux onto an existing linux file system
-distribution, just copy the Xen VMM (/boot/image.gz) and XenoLinux
-kernels (/boot/xenolinux.gz), then modify the Grub config
+If you're installing Xen/Xenlinux onto an existing linux file system
+distribution, just copy the Xen VMM (/boot/image.gz) and Xenlinux
+kernels (/boot/xenlinux.gz), then modify the Grub config
(/boot/grub/menu.lst or /boot/grub/grub.conf) on the target system.
It should work on pretty much any distribution.
A typical Grub menu option might look like:
-title Xen / XenoLinux 2.4.22
+title Xen / Xenlinux 2.4.22
kernel /boot/image.gz dom0_mem=131072 com1=115200,8n1 noht
- module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0
+ module /boot/xenlinux.gz root=/dev/sda4 ro console=tty0
The first line specifies which Xen image to use, and what command line
arguments to pass to Xen. In this case we set the maximum amount of
ifname=ethXX to select which one to use. If your network card is
unsupported, use ifname=dummy
-The second line specifies which xenolinux image to use, and the
+The second line specifies which xenlinux image to use, and the
standard linux command line arguments to pass to the kernel. In this
case, we're configuring the root partition and stating that it should
initially be mounted read-only (normal practice).
=======================================
If you haven't got an existing Linux installation onto which you can
-just drop down the Xen and XenoLinux images, then the file systems on
+just drop down the Xen and Xenlinux images, then the file systems on
the CD provide a quick way of doing an install. However, you're
probably better off in the long run doing a proper Redhat, Fedora,
Debian etc install rather than just doing the hack described below:
If you have a crash you'll likely get a crash dump containing an EIP
(PC) which, along with an 'objdump -d image', can be useful in
-figuring out what's happened. Debug a XenoLinux image just as you
+figuring out what's happened. Debug a Xenlinux image just as you
would any other Linux kernel.
We supply a handy debug terminal program which you can find in
'tools/xenctl/lib/console_client.py <server host> <server port>'
-Installing Xen / XenoLinux on a RedHat distribution
+Installing Xen / Xenlinux on a RedHat distribution
===================================================
-When using Xen / Xenolinux on a standard Linux distribution there are
+When using Xen / Xenlinux on a standard Linux distribution there are
a couple of things to watch out for:
The first Linux VM that is started when Xen boots start (Domain 0) is
/etc/rc.sysinit script #'ing the lines out of /etc/inittab won't work
as it ignores the '#' and tries to access them anyway.
-Every Xenolinux instance owns a bidirectional 'virtual console'.
+Every Xenlinux instance owns a bidirectional 'virtual console'.
The device node to which this console is attached can be configured
by specifying 'xencons=' on the OS command line:
'xencons=off' --> disable virtual console
scripts to be run in depending on the run level number passed on the
kernel command line.
-Xenolinux kernels can be built to use runtime loadable modules just
+Xenlinux kernels can be built to use runtime loadable modules just
like normal linux kernels. Modules should be installed under
/lib/modules in the normal way.
Description of how the XenDemoCD boots
======================================
-1. Grub is used to load Xen, a XenoLinux kernel, and an initrd (initial
+1. Grub is used to load Xen, a Xenlinux kernel, and an initrd (initial
ram disk). [The source of the version of Grub used is in /usr/local/src]
2. the init=/linuxrc command line causes linux to execute /linuxrc in
projects / xen.git / commitdiff